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, tearnitiy-in-work is an integral part of nxperionco Based Career Education Programs, 
intornships) cooperative and work-experience programs, and on-the-job components of vocational 
education. Since the early ^970s, there has been a movement in education to expand the education 
opportunities of all students to include "real world" learning experierices as part of the total f 
educational, experience. In an attempt tpjnvestigate the relationships of learning and work, tMe 
National Qenter for Research in Vocational Education has initiated a programmatic effort to conduct 
'basic research of the phenomena. This study, supported by the National Institute of Education, 
reports the findings of an exploratory examination of student retentiqn of mathematical and^ ^ 
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ABSTRACT ^ 

' r ' ' ' . ■ f 

■ . 

The study examined the retention of mathematical and reading concepts for students enrolled 
ip a learning-in-work ^nvirohment (Experience-Based Career Education) and a traditional classroom 
lOfjming environment on a measure of academic achievement in a twelve-month longitudinal design. 
5;tyafent performance in^each environment was evaluated using the Comprehensive Tests of 

Basic Skills, vvhich was admitiistered at tfje beginning and end of their junior year and at the 
beqinning of the senior year. Thus; the learning interval was designated as the time between pre- and 
post-testing, and the retention interval, the time between post- and foilow-up testing. The results 
indicated differences in both reading vs. math skills and in traditional vs. learning-in-work environ- 
ments.^^n interference/assimilation model was proposed to interpret the findings. 



Tfiis preliminary investigation is the first of a two-year effort. In the interest of parsimony 
v.and the wise use of resources, this report was prepared for dissemination to interested researchers 
[arid program'designers. For the selected few who may be interested, the larger document (175p) ' 
containing the technical information and appendices, is Mailable for inspection at the National 
Center. Finally, Appendix^A of this report contains the Table of ^ 

^ Contents, List of Figures, and List of Tables of the technical information'and appendices documen|. 
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PROBLEM ^ 

Since the early 1970s, there has beon an effort to expand students' educations! opportunities * 
to include a variety of "real-world" experiences in the workplace to complement the traditional ' 
classroom learning environmen t (Crowe «nd Adams, 1979). An example of a learning-in-work 
. program is the Experience-Based Career Education (EBCE) program. 

Briefly, the EBCE model is an academically-oriented, commu/iity-based program: students 
spend one day per week at the learning center with a leai-ning coordinator, who supervises and 
directs the learning activities of the students; fqur days per week are spent at a community- (work) 
site under the guidance and supervision of a resource persofj (worksite mentor). Students work at 
three to twelve sites per school year, depending on their career interests and academic needs. The 
program offers twenty-eight EBCE cqurses that are related to traditional subject matter disciplines. 
Students choose EBCE courses as a function of their assessment of (1 ) qareer interests and aptitudes 
(e.g., EBCE courses and occupations are related through the Dictionary of Occupational Titles 
worker trait groups) and (2) academic needs and interests (e.g., EBCE courses and traditional 
subject matter concepts are related through an instructional matrix that sets the parameters for 
designing the learning activities),/ - 

Under the EBCE model, academic development is accomplished through the use of an activity 
sheet. The activity sheets are designed to guide students' learning through a series of learning 
activities that relate subject matter concepts and cF.reer objectives to experiences at the work site. 
After concluding work on the activity sheets, students are evaluated by the learning coordinator and 
then plan future learning experiences. 

The EBCE approach would seem to offer the benefits of frequent, meaningful, and speedy 
feedback on task performance for students. Research in academic learning and retention (Ausubel, 
1968; Bdker, 1974; La Porte and Voss, 1975) would predict increased performance as a function of 
the above variables. Ausubel (1968) reported increased learning and retention of meaningful =vs. 
rote memorized material. Anderson and Biddle (.1975) and Boker.(1974) found a strong relationship 
between increased application (practice) of material and subsequent retention. Similarly, La Porte 
and Voss (1975) demonstrated superior retention performance as a f^jnction of usage of the 
information and response-contingent performance feedback. 

Given that past research using only a pre- test/post-test control group design has detected^few 
statistical differences in students' achievernents (Crowe and Adams, 1979; Crowe and Walker, 1977), ' 
tt was of interest in the present investiga^on to test the assumption that demonstrable changes in 
student performance may occur after participation in the prograpn. Thus, a retention model design 
was proposed using repeated measures whereby the learning inten/al was the time students participated 

% • 
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in the learnii^ enyirofirn^nt (tiino months) and the retention if»terval wa&ttje summer recess (three 
' months). If) viovv of previous resoarch on environmental contlncjencios as determinants of learmny 
and retention (see Gagne, 1978, for a review), it was predictod that students' exposure to different 
types of learning environments would affect performance on standardized tests of academic ability 
(i.e., CTBS). The study was exploratory in nature and exanuned performance in two learning 
environ«nef^ts lomjitudinally in order to detect changes in the retention of academic performance • 
which would result from parti^^ipation in one of the two environments during the learning interval. 

?• • ' » • 

METHOD 

Subjects 

Juniors from a suburban school district in Minnesota wore selected to participate In the study. 
This district was chosen by virtue of having both a traditional learning environment (classroom 
instruction) and learning-in-work environment (EBCii progranj) in operation. Twenty-seven students 
who volunteered for the EBCE program were successfully followed over the observation period. * 
Eleven students in the original EBCE sample' of thirty-eight werg dropped due to missing data at 
one or more of the subsequent testings. Control students in a jtradiiional environment were selected 
at random from a,oool of students to match the EBCE studentti on sex, school membership, and 
GPA. Twenty-seven control students^A/ere successfully followed over the observation period, while 
fifteen students were lost due to missing test data, 

i 

Measures 

» 

The CTBS Expanded Edition, Level 4, f^orm S was used- at all testings. Of the total test battery, 
Test 2 (Reading Comprehension) and Test 7 (Mathematics Concepts and Applications) were Selected 
for administration due to constraints on testing. The math test was scored to provide six scales, and 
the reading test, five scales. 

An additional scale to measure student perceptions of the complexity' of the learning environ- 
ment was developed (Learning Environment Questionnaire, LEQ). This scale was composed of 
thirteen items measuring the degree to which students perceived the environment as providing 
feedback, offering a variety of tasks, and giving direction to complete tasks. 

Students were interviewed by the researchers during May to gather further information on the 
instructional processes in the learning environments. 

a 

Proc^^Jure s 

The CTBS tests were administered on three occasions: pre-test, September, 1978; post-test, 
May, 1979; and follow-up test, September, 1979. The tests were group-administered to students in 
d school setting and were computer scored by the experimenters. The LEQ was given at the second 
tpsting, and interv;ew data was gathered the week of the post-test. 
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HliiSULTS 

Anatytiu Strategy • » ' . » 

♦ •. 

SincQ thb» purpose of i(\e pi«sor»t priiliminary irwesdyatiof} was ,to ideatify arous for futurt) 
in depth examination of the data, analysts of variance lachniqucs were used to evaluate the results ; 
of the reading and math CTBS scores. Given that the s^jmpie size wa(i relatively Smalt^ an attempt 
vvas made in alt analyses to use the smallest r nber of variables possible to allow higher statistical 
power to detect effec s which may have resent. A<:cordingiy, variables whicfi demonstrated 
insignificant effects in higher-order desig re deleted from subsequent analyses in an attempt 
to decrease Type II errors of inference. 

Three- Factor Anaiyses 

Repeated-measures analysis of variance (ANOVA) tj\t^ were performed on the results of the 
math scales (scale 1 -. content dimension, concepts; srdle 2<- content dimension, application; ' 
scale 3 - process dimension, recognition; scale 4 - proces:» dimension', translation; scale 5 - process 
dimension, interpretation; and scale 6 - process dimension, analysis). These scales constituted the 
dependent variables for each analysis. Independent variables were as follows: test - observations 
at J,, Tj, and T^ functioned as the repeated measure; program EBCE or treatment (T) group, 
tratiitional learners or control (C); and math instruction option - math class taken as a junior, 
no math class taken as a junior. The resultant design was defined as a 2 (programs) x 2 (math-no- 
math) X 3 (ttots) factorial, 

• t * 

%, 

If 

Results of these analyses are summarized in Table 1, which raveals significant main effects for 
tests on math scale 3 and significant tests by program interactions for scales 2, 3, and 5. No effects > 
' were seen for the math-no-r isth variable, which was therefore dropped from subsequent analyses 
(p<.05 reported for all analyses). 

Two- Factor Analyses 

Analyses of CTBS scores (math and reading) were then performed using a 2 (programs) x 3 (tests) 
design to allow increased power in detecting effects. The dependent variables for the reading compre- 
hension analyses consisted of the following five scales: scale 1 ~ content dimension, reading compre- 
hension; scale 2 - process dimension, recognition; scale 3 - process dimension, translation; scale 4 - 
process ditnerision, interpretation; and scale 5 - process dimension, analysis. Results of the repeated 
measures ANOVAs, analyses of simple effects, and LSD tests between. means are reported in Table 2. 
Main effects for program were absent in all analyses; main effects for tests were seen in reading, 
scale 1 and 4; tnteractiQns of programs by tests were seen in math scales 2, 3, and 5 and reading 
scale 5, These results are plotted in Figures 1 and 2 with scores expressed in raw units, Si.^ndard 
(feviation scores (SD^;>and qrade- equivalents (GE) when available. Figure 1 presents the results of 
ihf! math concepts aVialysis while Figure 5 presents the results bf the reading conqopts analysis. 

LEQ 

Comp.iMsons of v\v. resfjonses to t\w LEO it^ms U)\ trf.itmon' and ncmtrol students are reported 
•II T-ihtei 3 and 4 <ind were analy/er} via t t(!Sts. ' 
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FiGURE 1 

Student PierformanciB on Math Scales for Three Testings 
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FIGURE 2 

Student Performance cn Reading Comprehension SciUes fof Three Testings 
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t-TEST ANALYSIS FOR SCALE 1 OF LEQ 

/ 

LEARNING ENVIRONMENT OUESTIONNAIRE 

^ Hypothesized Scale 1 
Environmental Complexity 
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MEST ANALYSIS FOR SCALE 2 OF LE6^^^^- 

LEARNING ENVIRONMENT QUESTIONNAIRE 

Hypothesized Scale 2 - . 
Environmental Control 



f 0 



ITEM 




No. of 




Standard 

V IQ %iwll 


\ value 


Degrees 


TwoTall 
r rouauiiiiy 


2. i wa$ able to tftll by mys«lf 
if 1 was doino a aood iob. 


c 




3.73 


74 




77 


.0207 


0 • * 

i was ablo to tell by myself 
if t was doiny a good job. 


V T , 


38 


4.18 


.96 


2.36 


S. The results of what i did had 
meaning, i felt the results were 
important. 


C 


41 


3.66 


.91 • 


1 

. 3.45 

artJ 

i 


77 ' 

k 


..0009 


The results of what 1 did had 
meaning. 1 felt the results were 
important. . ^ . 




38 \. 


4.34 


.88 


8. The work that 1 did oVferedvine 
many different things to^. 


C 


^ 

41 


3.56 


.56 


* • 


77 


.0001 


— ' t^r^ 

The work that 1 did of f^d mT 
many different things to dd. 


T 


38 


4.45 


. 1.00 


'■\ 

11. The teachers orovided me 
opportunitij^to do meaniu||ifvil 
work or solve problemi. 


C 


41 


3.63 


.73 


- \^ 

9- 

« 

4.11 

% 


77 


.0001 


The resource person provided me 
ODDortunitiflt to rfo memiinAful 
work or solve problems. 


T 


' 38 


4.32 


.74 


The learning coordinator provided. < 
me opportunities to do meaningful 
work or solve problemi 


T 


38 


4.31 


.77 


1 

(d) 
4,01 


\ 

f7 


.0001 V 


13. The teachers encouraged me to 
decide for myself how 1 was going 
to do my work. 


1 

C 


41 


3.68 


1.01 


.23 


77, 


' .8195 

1 


The resource person encouraged 
me to decide for myself how 1 was 
going to do my work. 


T 


38 


3.74 


1.08 


The learnmg coordinator 
encouraged me to decide for 
myself how 1 was going to do 
my work. 


T 


38 


3.84 


1.10 


<d) 
.66 


77 


.5004 



(a) C ~ Control students in traditional learning environment 
T « Treatment students in learning in*work environment 

<bl t Test calculated for students who had both pre* and post test scores 

(e) t-Test comparing teachers of controls to resource persons of treatment students. 

(d) t Test comparing teachers of controls to learning coordinators of tfg4tment students. 
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Summer Activities 



Studunts reported their summer activities on tho third-questionnaire, which were then 
class.f.ed.ns follows: part-time paid omploymont - 11 HBCEi, 16 control; full-time paid employ- 
ment 9 fcBQ£, 5 control; "other" 5 BBCB, 3 control;,^nd summer school plus part-time paid 
empfoyment 2 EBCE, 3 control.' Results of chi-square af<i!lv4on the 2 (treatment-control) x 4 
(summer act.vity classes) frequency matrix {Xj = 4.2. df = 4),v= 0.38) indicated no significant 
association^ betwuen these factors. • * ■ 



DISCUSSION 

LEQ and Interviews 

\ 

^ While the.researchers recbgnize.that the L6Q is currently in a developmental stage, the results 
of both the LEQ and the interviews with students were seen to indicate that students in the 
alternative (learning-in-work) environment perceived:' (1). s greater chance to find things out on 
their own.(autonomy); (2) more support from instructors for developing their own ideas; (3) more 
feedback 'on their performance (self -generated); (4) more variety in their "learning tasks; and (5) 
more meaning in their activities.^ Overall these are similar characteristics to those identified by 
research on learning and retention of material (Gagne, 1978) that lead to increased performance on 
learning tasks. It was concluded from these data that the learning-in-work environment does con- 
tribute to conditions favorable for academic learning and>etention. ■/ 

■ ■ ' / ■ 

CI BS Scales: Reading Comprehension 

The effect of these two environments on standardized scholastic achievement performance 
was evaluated by the'ANOVAs on the CTBS data. Rese^lts of reading scores indicated that, in general, 
both groups Remonstrated equal performance that increased linearly for reading scales 1 -4 over the 
learning interval (pre-test to post-test)' and over the retention interval (post-test to follow-up). 
Significant mean increases were seen only for reading scales 1 and 4, comparing pre-test to follow-up. 
Groups did pot differ significantly ("[ vs. C) at any observation, on any reading scale, indicating a 
similar level of performance on these measures. ^ 

CTBS Scales: Mathematics .Concepts and Application 

ft Inspection of the math scores reveals a different patfern for the groups. Three of the six math 
scales (2, 3, and 5) revealed significant interactions of program (T vs. C) by Tests, indicating, 
significantly nonparallel learning/retention functi9ns. Additionally, the two groups differed' 
significantly at the end ofnhe school year but were nonsignificantly different at the start.of their 
junior year and ai the start of their senior year. Thus, groups were equivalent at the point where 
they were split into different learning environments, but the students in the traditional learning 
environment increased in performance up to the end of Jhe year, while the learning-in-work students 
showed a decrease in performance on the post-test. This effect was manifest in scales 3 and 5 at 
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p< m. whilo suilHS 1, 2, and 4 sltowyU trends in this direction. The group? subsequently revorsed 
this direction of change over the summer: the students in the traditional learning environment' . 
doavased from post- to follow-up test, while learning-in-work students incteasiid over the sumrxior. 
The Qvorall composite form of this relationship dcross matf. tests is depicted in Figure 3. 

Interprototion 

These findings were not predicted -from previoys research, which would sugaest an increase 
for both groups during the instructional period, and a "Wgetting" gradient over'^the su^timer 
(Ebbinghaus, 1885). Alternative posf/7oc explanations for these results were considered 6y the. 
authors, which centered on uncontrolled effects of student selection for the learning-in-work 
proyrdm. While it was deemed possible that, due to nomaridom assignment of students to the 
EBCE program, subjects may have systematically differed on a variable that could interact with 
the repeated testings, this explanation was not favored in view of the lack of.dilferences between 
(jroups at the start of the study. Additionally, a mechanism would be necessary to account for the 
unequal.effect of such a confounded selection, such that it \A/ould^have no effect at test 1, would 
prpduce a significant decrement at test 2, and would then increase performance for the EBCE 
students at the same time that controls (in the same environment) were decreasing over the summer. 
This necessarily unparsimonious approach was rejected in interpreting the results of the math- 
performance. 

I 

.. A model was proposed to account for the unequal learning and retention of CTBS math skfl'ls, 
which emphasized two constructs: interference due. to previous learning (retroactive interference) ' 
and assimilation of information. This model is displayed in Figure 4. On the basis of both interview 
and LEQ data, the researchers hypothesized that students in the l4rning-in-work environment 
were learning different things than the controls in a traditional setting. Specifically, it was^postulated 
that EBCE students were learriing newi'rules" for learning how to4earn or function in a work 
environment. These rules for learning l^ow to learn were seen to be different than those typically • 
measured by academic tests of achievement (e.g., CTBS). The paradigm is as follows: (1) students 
in both groups have similar learning histories up to the first testing, which emphasized (raditional, 
nonapplied, use of math* constructs; ,(2) students placed in the learning-in-work environn^e^t then 
w^e forced to use or generate constructs in an applied setting, and to generalize from abstract math 
concepts to applied math usage on the fobs; (3) traditionally learned math skills were not being 
practiced on the |ob, and new ways to use math were being learned which did not necessarily 
overlap with previous learning; (4) these new rules for learning were not assimilated into the previous 
math framework or structure learned in class, and constituted a retroactive inhibitor to the retrieval 
of the CTBS measured math skills which emerged as a function of dissimilarities for the two ways 
of learning matfi (measures at Tj ); and (5) follow-up performance on CTBS was seen to depend on 
the extent to new ways to use math were assimilated into the previous framework or 

structure of math knowledge. That is, CTBS performance should increase aS a function of^the - 
degree to which students could relate the new rules for using math to the way math is measured 
f»V the CTBS. \ 
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FIGUR£E4 

Interpretation of Results From A Retroactive'Design Perspective 
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Conv;erse!v. it was predicted that the use of a performancu-based math test (content-valid fo^ 
applied settings) would reveal that leacning-in work students wore not "losing information" but ( 
were learning nawrulea for learning how to learn wtsich wore not manifest on the CTBS until they 
could b'e assimilated into^the previously developed traditional structure for remembering tnath 
concepts-. Ai^coraingiy, future research should measure the effects of different jeai'hing environments 
by employing :ctassroom-t}ased tests tests which are content-valid for measuring tnath skills iis 
they exist in the work environment. This type of design would allow a ppwerful'test of the model's 
predictions regarding perfbrmanc'e on subject matter concepts m different learning environments, 
and could further illustrate the merjt of hands-on learning as.is offered by learning-in-work 
environments. s 
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